Three-Dimensional Image Processing Apparatus, Three-Dimensional Image Processing Method, Three-Dimensional Image Processing Program, Computer-Readable Recording Medium, And Recording Device

ABSTRACT

A three-dimensional image processing apparatus includes: a distance image generating part capable of generating a distance image based on a plurality of images captured in an image capturing part; a pattern generating part for generating a first projection pattern and a second projection pattern whose fringe direction is different from that of the first projection pattern as a plurality of projection patterns obtained by changing a fringe direction of a projection pattern; and an incorrect-height determining part for making comparison in height information of a corresponding portion of an inspection target between a first distance image, generated in the distance image generating part based on a first pattern projected image, and a second distance image, generated based on a second pattern projected image, to determine height information of a portion where a difference not smaller than a predetermined value has occurred as incorrect.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims foreign priority based on Japanese PatentApplication No. 2013-216850, filed Oct. 17, 2013, the contents of whichis incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a three-dimensional image processingapparatus, a three-dimensional image processing method, athree-dimensional image processing program, a computer-readablerecording medium, and a recording device.

2. Description of Related Art

In a large number of production sites such as factories, there have beenintroduced image processing apparatuses that realize automatic and fastperformance of inspections, which have relied on viewing of humans. Theimage processing apparatus captures an image of a workpiece that comesflowing through a production line such as a belt conveyor by use of acamera, and executes measurement processing such as edge detection andarea calculation for a predetermined region by use of the obtained imagedata. Then, based on a processing result of the measurement processing,the apparatus performs inspections such as detection of a crack on theworkpiece and positional detection of alignment marks, and outputsdetermination signals for determining the presence or absence of a crackon the workpiece and positional displacement. In such a manner, theimage processing apparatus may be used as one of FA (Factory Automation)sensors.

An image which is taken as a measurement processing target by the imageprocessing apparatus that is used as the FA sensor is principally abrightness image not including height information. For this reason,speaking of the foregoing detection of a crack on the workpiece, theapparatus is good at stably detecting a two-dimensional shape of acracked portion, but having difficulties in stably detecting athree-dimensional shape of, for example, a depression of a flaw which isnot apt to appear in a brightness image. For example, it is thought thata type or a direction of illumination that illuminates the workpieceduring the inspection is devised and a shade caused by a depression of aflaw is detected to indirectly detect a three-dimensional shape, but aclear shade is not necessarily always detected in the brightness image.In order to prevent an erroneous determination which is to erroneouslydetect a defective product as a non-defective product at the time of anunclear shade being detected, for example, if a determination thresholdis biased to the safe side, the apparatus might determine a large numberof non-defective products as defective products, to cause deteriorationin production yield.

Accordingly, there is considered a visual inspection which uses not onlya shade image that takes, as a pixel value, a shade value in accordancewith a light reception amount of the camera but also a distance imagethat takes, as a pixel value, a shade value in accordance with adistance from the camera to the workpiece to two-dimensionally express aheight (e.g. see Unexamined Japanese Patent Publication No. 2012-21909).

An example of the three-dimensional image processing apparatus is shownin a schematic view of FIG. 23. This three-dimensional image processingapparatus 190 is configured of a head section 191 provided with an imagecapturing part such as a light reception element, and a controllersection 192 which is connected to the head section 191 and is sent imagedata captured in the head section 191, to generate a distance image fromthe acquired image data.

Here, the principle of triangulation will be described based on FIG. 23.In the head section 191, an angle α between an optical axis of incidentlight emitted from a light projecting section 110 and an optical axis ofreflected light that is incident on a light receiving section 120(optical axis of the light receiving section 120) is previously set.Here, when the workpiece WK is not mounted on a stage 140, incidentlight emitted from the light projecting section 110 is reflected by apoint O on the workpiece WK mounting surface and is incident on thelight receiving section 120. On the other hand, when the workpiece WK ismounted on the stage 140, the incident light emitted from the lightprojecting section 110 is reflected by a point A on the surface of theworkpiece WK and is incident as reflected light on the light receivingsection 120. Then, a distance d in an X-direction between the point Oand the point A is measured, and based on this distance d, a height h ofthe point A on the surface of the workpiece WK is calculated.

Heights of all points on the surface of the workpiece WK are calculatedapplying the foregoing the measurement principle of triangulation,thereby to measure a three-dimensional shape of the workpiece WK. In apattern projecting method, in order that all the points on the surfaceof the workpiece WK are irradiated with incident light, the incidentlight is emitted from the light projecting section 110 in accordancewith a predetermined structured pattern, reflected light as the lightreflected on the surface of the workpiece WK is received, and based on aplurality of received pattern images, the three-dimensional shape of theworkpiece WK is efficiently measured.

As such a pattern projecting method, there are known a phase shiftmethod, a spatial coding method, a multi-slit method and the like. Bythe three-dimensional measurement processing performed using the patternprojecting method, a projection pattern is changed to repeatimage-capturing a plurality of times in the head section, and the imagesare transmitted to the controller section. In the controller section,computing is performed based on the pattern projected images transmittedfrom the head section, and a distance image having height information ofthe workpiece can be obtained.

In 3D measurement using the phase shift method and the spatial codingmethod by the configuration of the camera and the projector as thusdescribed, a fringe pattern as shown in FIG. 24 is projected as aprojection pattern PN. However, in the pattern projecting method, therehas been a problem in that a measurement result which is incorrect inprinciple is obtained in a place where multi-reflection occurs. As shownin FIG. 25, especially on the side surface of the workpiece WK whichstands upright as seen from the camera, measurement becomes incorrectdue to an influence of mirror reflection, and post-stage processing isaffected by mixture of an effective result and an incorrect result.

However, since an image of a mirror-reflected pattern is captured as aclear pattern, differently from the case of deficiency in light amount,it is displayed as a clear measurement result similar to a normalmeasurement result. It has hitherto been difficult to determine whetherthe result is abnormal and disadvantageously a different result from theoriginal has been obtained.

Specifically, as in FIG. 26, there will be considered an example wherean image of workpieces WKA, WKB and WKC placed on a stage ST for placingthe workpiece WK is captured by means of a camera as an image capturingpart 10 and a projector as a light projecting part 20. In this case, theworkpieces WKA, WKB and WKC as seen from the camera are as in FIG. 27.In this state, when a projection pattern PN0 as in FIG. 24 is projectedto the workpieces WKA, WKB and WKC, an image of a fringe patterncaptured in the camera is as in FIG. 28. Among them, the fringe patternsobtained by capturing the workpiece WKB and the workpiece WKC are as inan enlarged view of FIG. 29. Here, since the side surfaces of theworkpiece WKB and the workpiece WKC have acute angles as shown in FIG.25, each of them comes into a state where a pattern on the bottomsurface is reflected thereon, namely the mirror reflection occurs (inFIG. 29, foreign substances are displayed for facilitating understandingof reflection of mirror patterns due to the mirror reflection). As aresult, there is obtained a pattern different from a pattern that shouldoriginally be obtained in a state where the mirror reflection does notoccur. Specifically, an original distance image to be measured withrespect to the workpieces of FIG. 27 should be as in FIG. 30A, but as aresult of capturing of the image of the fringe pattern as in FIG. 28, anincorrect distance image as in FIG. 30B is measured. It is to be notedthat in these drawings, a region which is shaded by the workpiece andwhose height is thus non-measurable is shown by being painted out inblack. Further, a brighter place (a place closer to white) indicates ithas a larger height, and a darker place indicates it has a smallerheight.

As apparent by comparing FIG. 30A and FIG. 30B, a normal height has beenmeasured concerning the workpiece WKA. On the other hand, as for theworkpiece WKB, the top surface has been correctly measured since nomirror reflection occurs, but the front-side surface has been computedas incorrect height information as if it is continuous with the floorsurface of the stage ST at the same height. Further, as for theworkpiece WKC, the top surface has been correctly measured as well, butan incorrect measurement result has been obtained for the side surfacefacing lower left as if it further goes under the floor surface of thestage.

The reason why such incorrect results are computed is that, as a resultof occurrence of the mirror reflection, a projection pattern differentfrom a projection pattern that should originally be obtained is measuredand a result of height calculation by means of triangulation thusbecomes incorrect. Specifically, in the case of the workpiece WKB, asshown in FIG. 31C, the fringe pattern is projected from an obliquedirection by the projector to the side surface of the workpiece WKB, andhence height information is acquired at an intersection with a visualline of an observation point. When such a fringe pattern is projected tothe workpiece WKB, an original normal projection pattern is as in FIG.31A. However, as a result of occurrence of the mirror reflection, apattern on the floor surface of the stage is reflected on the sidesurface, and a projection pattern as in FIG. 31B is obtained. Forexample, when a focus is placed on an observation point P₁ on the sidesurface of FIG. 31A, it should originally be irradiated with a fringe 3,but it actually looks as if being irradiated with a fringe 2 as in FIG.31B. Consequently, as in FIG. 31C, height information of the sidesurface of the workpiece WKB is detected not as one at an intersectionof the visual line of the observation point with the fringe 3 which hasan original height, but as one at an intersection with the fringe 2,resulting in that the side surface is measured as having the same heightas that of the floor surface of the stage.

Further, in the case of the workpiece WKC, as shown in FIG. 32C, whenthe fringe pattern is projected from an oblique direction from theprojector to the side surface of the workpiece WKC, an original normalprojection pattern should be as in FIG. 32A, but a projection pattern asin FIG. 32B is obtained due to the mirror reflection. In this example,when a focus is placed on an observation point P₂, it should originallybe irradiated with a fringe 5, but it actually looks as if beingirradiated with a fringe 4 as in FIG. 32B. Consequently, as in FIG. 32C,height information of the side surface of the workpiece WKC is detectednot as one at an intersection of a visual line of the observation pointwith the fringe 5 which has an original height, but as one at anintersection with the fringe 4, resulting in that the side surface ismeasured as having a smaller height than that of the floor surface ofthe stage.

As thus described, in a portion where the mirror reflection hasoccurred, it is not possible to distinguish between a normal measurementresult and an incorrect measurement result, thus causing a problem ofaffecting the post-stage processing.

SUMMARY OF THE INVENTION

The present invention is for solving the conventional problems asdescribed above. A principal object of the present invention is toprovide a three-dimensional image processing apparatus, athree-dimensional image processing method, a three-dimensional imageprocessing program, a computer-readable recording medium, and arecording device, each being capable of suppressing a measurement errordue to multi-reflection.

For achieving the above object, a three-dimensional image processingapparatus according to one embodiment of the present invention is athree-dimensional image processing apparatus, which is capable ofacquiring a distance image that includes height information of aninspection target and also performing image processing based on thedistance image. The apparatus can include: a light projecting part forprojecting incident light as structured illumination of a predeterminedfringe-like projection pattern from an oblique direction with respect toan optical axis of a below-described image capturing part; the imagecapturing part for acquiring reflected light that is projected by thelight projecting part and reflected on an inspection target, to capturea plurality of pattern projected images; a distance image generatingpart capable of generating a distance image based on the plurality ofimages captured in the image capturing part; a pattern generating partfor generating a first projection pattern and a second projectionpattern whose fringe direction is different from that of the firstprojection pattern as a plurality of projection patterns obtained bychanging a fringe direction of the projection pattern; and anincorrect-height determining part for making comparison in heightinformation of a corresponding portion of the inspection target betweena first distance image, generated by the distance image generating partbased on a first pattern projected image obtained by projecting from thelight-projecting part the first projection pattern generated in thepattern generating part and capturing an image of the first projectionpattern in the image capturing part, and a second distance image,generated based on a second pattern projected image obtained by thesecond projection pattern projected from the common light projectingpart, to determine height information of a portion where a differencenot smaller than a predetermined value has occurred as incorrect. Withthe above configuration, a portion where incorrect height informationhas occurred due to mirror reflection or the like can be detected, whichhas hitherto been difficult, thereby to allow improvement in reliabilityof a measurement result.

Further, in a three-dimensional image processing apparatus according toanother embodiment, it can be configured such that the patterngenerating part generates as the first projection pattern a projectionpattern whose fringe direction is inclined with respect to a verticaldirection and a horizontal direction.

Further, in a three-dimensional image processing apparatus according toanother embodiment, it can be configured such that the patterngenerating part generates as the first projection pattern a projectionpattern whose fringe direction is inclined by 45° from the verticaldirection.

Further, in a three-dimensional image processing apparatus according toanother embodiment, it can be configured such that the patterngenerating part generates as the second projection pattern a projectionpattern that is symmetrical to the first projection pattern.

Further, in a three-dimensional image processing apparatus according toanother embodiment, the pattern generating part can include a digitalmicro-mirror device.

Further, in a three-dimensional image processing apparatus according toanother embodiment, a micro-mirror constituting each pixel of thedigital micro-mirror device has a rectangular shape in a plan view, andeach micro-mirror can be arranged in a posture inclined in a diamondshape. With the above configuration, there can be obtained an advantageof reducing the roughness of the fringes at the time of projecting theprojection pattern inclined diagonally.

Further, in a three-dimensional image processing apparatus according toanother embodiment, it can be configured such that the distance imagegenerating part is capable of generating a synthetic image obtained byaveraging the first distance image and the second distance image exceptfor a portion determined as incorrect in the incorrect-heightdetermining part. With the above configuration, there can be obtained anadvantage of being able to generate a distance image where a portionincluding incorrect height information is removed while the measurementaccuracy in the other region is further improved.

Further, in a three-dimensional image processing apparatus according toanother embodiment, it can be configured such that the incorrect-heightdetermining part inserts a flag as incorrect information into a portionwhere a difference not smaller than the predetermined value isdetermined to have occurred. With the above configuration, at the timeof performing computing by use of height information, a portion with anincorrect value can be readily specified, to allow processing notaffected by the incorrect result.

Further, in a three-dimensional image processing apparatus according toanother embodiment, the incorrect-height determining part can treat aportion where a difference not smaller than the predetermined value isdetermined to have occurred, in a similar manner to a non-measurableplace. With the above configuration, at the time of performing computingby use of height information, a portion with an incorrect value can betreated in a similar manner to a place where phase-shift contrast is lowdue to a shade and measurement cannot be performed, or some other place.

Further, in a three-dimensional image processing apparatus according toanother embodiment, the light projecting part can project structuredillumination for obtaining the distance image by use of at least a phaseshift method and a spatial coding method.

Further, a three-dimensional image processing method according toanother embodiment is a three-dimensional image processing method foracquiring a distance image that includes height information of aninspection target and performing image processing based on the distanceimage. The method can include the steps of: previously setting a firstprojection pattern and a second projection pattern whose fringedirection is different from that of the first projection pattern as aplurality of projection patterns obtained by changing a fringe directionof a projection pattern for structured illumination, and projecting froma light projecting part incident light as structured illumination of thefirst projection pattern from an oblique direction with respect to anoptical axis of an image capturing part; acquiring reflected light thatis projected by the light projecting part and reflected on an inspectiontarget, to capture a first pattern projected image in the imagecapturing part; generating a first distance image in a distance imagegenerating part based on a plurality of images captured in the imagecapturing part; projecting from the light projecting part incident lightas structured illumination of the second projection pattern from thesame direction as the first projection pattern with respect to theoptical axis of the image capturing part; acquiring reflected light thatis projected by the light projecting part and reflected on theinspection target, to capture a second pattern projected image in theimage capturing part; generating a second distance image in the distanceimage generating part based on a plurality of images captured in theimage capturing part; and making comparison in height information of acorresponding portion of the inspection target between the firstdistance image and the second distance image, to determine heightinformation of a portion where a difference not smaller than apredetermined value has occurred as incorrect if this exists. Herewith,a portion where incorrect height information has occurred due to mirrorreflection or the like can be detected, which has hitherto beendifficult, thereby to allow improvement in reliability of a measurementresult.

Further, a three-dimensional image processing program according toanother embodiment is a three-dimensional image processing program,which is capable of acquiring a distance image that includes heightinformation of an inspection target and also performing image processingbased on the distance image. The program can allow a computer torealize: a projection pattern setting function of previously setting afirst projection pattern and a second projection pattern whose fringedirection is different from that of the first projection pattern as aplurality of projection patterns obtained by changing a fringe directionof a projection pattern for structured illumination; a distance imagegenerating function of generating a first distance image and a seconddistance image based on a first pattern projected image and a secondpattern projected image captured by projecting by a light projectingpart incident light as structured illumination of the first projectionpattern and the second projection pattern set by the projection patternsetting function, from an oblique direction with respect to an opticalaxis of an image capturing part and acquiring reflected light reflectedon an inspection target; and an incorrect-height determining function ofmaking comparison in height information of a corresponding portion ofthe inspection target between the first distance image and the seconddistance image, to determine height information of a portion where adifference not smaller than a predetermined value has occurred asincorrect if this exists. With the above configuration, a portion whereincorrect height information has occurred due to mirror reflection orthe like can be detected, which has hitherto been difficult, thereby toallow improvement in reliability of a measurement result.

Further, a computer-readable recording medium or a storage deviceaccording to another embodiment is one in which the three-dimensionalimage processing program is to be stored. The recording medium includesa magnetic disk, an optical disk, a magneto-optical disk, asemiconductor memory, and other program-storable medium, such as aCD-ROM, a CD-R, a CD-RW, a flexible disk, a magnetic tape, an MO, aDVD-ROM, a DVD-RAM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a Blu-ray(registered trademark), and an HD-DVD (AOD). Further, the programincludes one in the form of being distributed by downloading through anetwork such as the Internet, in addition to one stored into the aboverecording medium and distributed. Moreover, the storage device includesgeneral-purpose or a special-purpose device mounted with the program inthe form of software, firmware or the like, in an executable state.Furthermore, each processing and each function included in the programmay be executed by program software that is executable by the computer,and processing of each section may be realized by predetermined hardwaresuch as a gate array (FPGA, ASIC) or in the form of program softwarebeing mixed with a partial hardware module that realizes some element ofhardware.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view showing a system constitutional example of athree-dimensional image processing system including an image processingapparatus according to an embodiment of the present invention;

FIG. 2 is a view showing a system constitutional example of athree-dimensional image processing system according to a modifiedexample of the present invention;

FIG. 3 is a schematic view showing a hardware configuration of athree-dimensional image processing apparatus according to a secondembodiment of the present invention;

FIG. 4A is a head section of a three-dimensional image processingapparatus according to a third embodiment of the present invention, andFIG. 4B is a schematic view showing a head section of athree-dimensional image processing apparatus according to a fourthembodiment;

FIG. 5 is a block diagram showing a three-dimensional image processingapparatus according to the third embodiment of the present invention;

FIG. 6 is a plan view showing a DMD arranged with micro-mirrors eachformed in a diamond shape;

FIG. 7 is an enlarged plan view showing the micro-mirrors thatconstitute the DMD of FIG. 6;

FIG. 8 is a plan view showing a DMD arranged with micro-mirrors in agrid pattern;

FIG. 9 is a block diagram showing a controller section of FIG. 5;

FIG. 10 is a flowchart showing a processing operation of the imageprocessing apparatus according to the present embodiment;

FIG. 11 is a block diagram showing three-dimensional image processingaccording to an embodiment of the present invention;

FIG. 12 is an image view showing a first projection pattern;

FIG. 13 is an image view showing a second projection pattern;

FIG. 14 is a flowchart showing a procedure for executingincorrect-height determination;

FIG. 15A is an image view showing a first projection pattern image withrespect to workpieces of FIG. 27, and FIG. 15B is an image view showingthe first projection pattern image;

FIG. 16A is an image view showing a second projection pattern image withrespect to the workpieces of FIG. 27, and FIG. 16B is an image viewshowing the second projection pattern image;

FIG. 17A shows an image view of an original distance image of theworkpieces of FIG. 27, FIG. 17B shows an image view of a distance imageobtained from the first projection pattern image, and FIG. 17C shows animage view of a distance image obtained from the second projectionpattern image;

FIG. 18A is an original first projection pattern image of a workpieceWKB, FIG. 18B is a first projection pattern image obtained due to mirrorreflection, and FIG. 18C is a schematic view showing measured heightinformation;

FIG. 19A is an original second projection pattern image of the workpieceWKB, FIG. 19B is a second projection pattern image obtained due to themirror reflection, and FIG. 19C is a schematic view showing measuredheight information;

FIG. 20A is an original first projection pattern image of a workpieceWKC, FIG. 20B is a first projection pattern image obtained due to themirror reflection, and FIG. 20C is a schematic view showing measuredheight information;

FIG. 21A is an original second projection pattern image of the workpieceWKC, FIG. 21B is a second projection pattern image obtained due to themirror reflection, and FIG. 21C is a schematic view showing measuredheight information;

FIG. 22A shows an image view of a first distance image of the workpiecesof FIG. 27, FIG. 22B shows an image view of a second distance image,FIG. 22C shows an image view of an image obtained by extracting areaswith incorrect height information from FIGS. 22A and 22B, and FIG. 22Dshows an image view of a synthetic distance image obtained by removingFIG. 22C from FIGS. 22A and 22B and performing averaging;

FIG. 23 is a schematic view showing a situation where a distance imageis captured by a triangulation system;

FIG. 24 is an image view showing an example of a fringe pattern;

FIG. 25 is a schematic view showing a situation where a portion of aworkpiece which has an acute angle is observed;

FIG. 26 is a schematic view showing a positional relation among theworkpieces, a camera and a projector;

FIG. 27 is a schematic view showing a state where the workpieces areseen from the camera of FIG. 26;

FIG. 28 is an image view showing a projection pattern image obtained byprojecting the projection pattern of FIG. 24 to the workpieces of FIG.27;

FIG. 29 is an enlarged view showing the workpiece WKB and the workpieceWKC of the projection pattern image of FIG. 28;

FIG. 30A is an image view showing a distance image that shouldoriginally be obtained with respect to the workpieces of FIG. 27, andFIG. 30B is an image view showing an incorrect distance image obtaineddue to the mirror reflection;

FIG. 31A is a normal projection pattern with respect to the workpieceWKB of FIG. 27, FIG. 31B is a projection pattern where the mirrorreflection has occurred with respect to the workpiece WKB, and FIG. 31Cis a schematic view showing measured height information; and

FIG. 32A is a normal projection pattern with respect to the workpieceWKC of FIG. 27, FIG. 32B is a projection pattern where the mirrorreflection has occurred with respect to the workpiece WKC, and FIG. 32Cis a schematic view showing measured height information.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Hereinafter, embodiments of the present invention will be describedbased on the drawings. However, the embodiments shown hereinafter areones illustrating a three-dimensional image processing apparatus, astate-change determining method for a three-dimensional image processingapparatus, a state-change determining program for a three-dimensionalimage processing apparatus, a computer-readable recording medium and arecording device for the purpose of embodying technical ideas of thepresent invention, and the present invention does not specify, to thefollowing, the three-dimensional image processing apparatus, thestate-change determining method for the three-dimensional imageprocessing apparatus, the state-change determining program for thethree-dimensional image processing apparatus, the computer-readablerecording medium and the recording device. Further, the presentspecification is not to specify members shown in the claims to membersof the embodiments. Especially, sizes, materials, shapes, relativedisposition and the like of constituent components described in theembodiments are not intended to restrict the scope of the presentinvention thereto, but are mere explanatory examples. It is to be notedthat sizes, positional relations and the like of members shown in eachdrawing may be exaggerated for clarifying a description. Further, in thefollowing description, the same name and symbol denote the same memberor members of the same quality, and a detailed description thereof willbe omitted as appropriate. Moreover, each element constituting thepresent invention may have a mode where a plurality of elements areconfigured of the same member and the one member may serve as theplurality of elements, or conversely, a function of one member can beshared and realized by a plurality of members.

Further, when a “distance image (height image) is referred to in thepresent specification, it is used in the meaning of being an imageincluding height information, and for example, it is used in the meaningof including in the distance image a three-dimensional synthesized imageobtained by pasting an optical brightness image to the distance image astexture information. Moreover, a displayed form of the distance image inthe present specification is not restricted to one displayed in atwo-dimensional form, but includes one displayed in a three-dimensionalform.

First Embodiment

FIG. 1 shows a configuration of a three-dimensional image processingapparatus according to a first embodiment of the present invention. Thisthree-dimensional image processing apparatus 100 is provided with a headsection 1 and a controller section 2. The head section 1 is providedwith a light projecting part 20 for illuminating an inspection target(workpiece) WK, an image capturing part 10 for capturing an image of theworkpiece WK, and a head-side communication part 36 for connecting withthe controller section 2.

On the other hand, the controller section 2 executes measurementprocessing such as edge detection and area calculation based on thecaptured image. Moreover, the controller section 2 can be detachablyconnected with a display part 4 such as a liquid crystal panel, an inputpart 3 such as a console for a user performing a variety of operationson the display part 4, a PLC (Programmable Logic Controller), and thelike.

The above three-dimensional image processing apparatus 100 projectsmeasurement light to the workpiece WK by the light projecting part 20 ofthe head section 1, and reflected light which has been incident andreflected on the workpiece WK is captured as a pattern projected imagein the image capturing part 10. Further, a distance image is generatedbased on the pattern projected image, and this distance image is furtherconverted to a low-tone distance image obtained by replacing heightinformation in each pixel with brightness. The controller section 2executes measurement processing such as edge detection and areacalculation based on the converted low-tone distance image.

It is to be noted that the workpiece WK as an inspection target is, forexample, an article which is sequentially carried on a production line,and is moving or standing still. Further, the moving workpiece includesone that rotates, in addition to one that moves by means of, forexample, a conveyor.

(Light Projecting Part 20)

The light projecting part 20 is used as illumination that illuminatesthe workpiece WK for generating the distance image. Therefore, the lightprojecting part 20 can, for example, be a pattern projector or the likefor projecting a sinusoidal fringe pattern to the workpiece inaccordance with a pattern projecting method for acquiring the distanceimage. Further, in addition to the light projecting part, a generalillumination apparatus for performing bright field illumination or darkfield illumination may be separately provided. Alternatively, it is alsopossible to allow the light projecting part 20 to have a function as thegeneral illumination apparatus.

The controller section 2 executes image processing by use of distanceimage data acquired from the head section 1, and outputs a determinationsignal as a signal indicating a determination result for thedefectiveness/non-defectiveness of the workpiece, or the like, to acontrol device such as an externally connected PLC 70.

The image capturing part 10 captures an image of the workpiece based ona control signal that is inputted from the PLC 70, such as an imagecapturing trigger signal that specifies timing for fetching image datafrom the image capturing part 10.

The display part 4 is a display apparatus for displaying image dataobtained by capturing the image of the workpiece and a result ofmeasurement processing by use of the image data. Generally, the user canconfirm an operating state of the controller section 2 by viewing thedisplay part 4. The input part 3 is an input apparatus for moving afocused position or selecting a menu item on the display part 4. Itshould be noted that in the case of using a touch panel for the displaypart 4, it can serve as both the display part and the input part.

Further, the controller section 2 can also be connected to a personalcomputer PC for generating a control program of the controller section2. Moreover, the personal computer PC can be installed with athree-dimensional image processing program for performing a settingconcerning three-dimensional image processing, to perform a variety ofsettings for processing that is performed in the controller section 2.Alternatively, by means of software that operates on this personalcomputer PC, it is possible to generate a processing sequence programfor prescribing a processing sequence for image processing. In thecontroller section 2, each image processing is sequentially executedalong the processing sequence. The personal computer PC and thecontroller section 2 are connected with each other via a communicationnetwork, and the processing sequence program generated on the personalcomputer PC is transferred to the controller section 2 along with, forexample, layout information for prescribing a display mode of thedisplay part 4, or the like. Further, in contrast, the processingsequence program, layout information and the like can be fetched fromthe controller section 2 and edited on the personal computer PC.

It is to be noted that this processing sequence program may be madegenerable not only on the personal computer PC but also in thecontroller section 2.

Modified Example

It is to be noted that, although the dedicated hardware is constructedas the controller section 2 in the above example, the present inventionis not restricted to this configuration. For example, as in athree-dimensional image processing apparatus 100′ according to amodified example shown in FIG. 2, one formed by installing a dedicatedinspection program or the three-dimensional image processing programinto a general-purpose personal computer, a work station or the like canbe functioned as a controller section 2′ and used as connected to thehead section 1. This three-dimensional image processing apparatusperforms a necessary setting for the image processing and the like bymeans of the three-dimensional image processing program, and thereafterperforms the image processing on the low-tone distance image inaccordance with the pattern projected image captured in the head section1, to perform a necessary inspection.

(Head-Side Communication Part 36)

Further, in accordance therewith, such an interface as to be connectedto either the dedicated controller section 2 or the personal computerthat functions as the controller section 2 can also be provided as thehead-side communication part 36 on the head section 1 side. For example,the head section 1 is provided with, as the head-side communication part36, a controller connecting interface 36A for connecting with thecontroller section 2 as shown in FIG. 1, or a PC connecting interface36B for connecting with the personal computer as shown in FIG. 2.Further, if such an interface is formed in a unit type so as to bereplaceable, other configurations of the head section are made common toa certain extent, and the common head section can thereby be connectedwith either the controller section or the personal computer.Alternatively, there can be provided one head-side communication partprovided with an interface connectable with either the dedicatedcontroller section 2 or the personal computer. Further, as such aninterface, an existing communication standard such as Ethernet (productname), a USB or RS-232C may be used. Moreover, a prescribed orgeneral-use communication system is not necessarily applied, but adedicated communication system may be applied.

(PC Connection Mode)

Further, the three-dimensional image processing program can be providedwith a PC connection mode for performing a setting in the case of usingthe personal computer as the controller section 2′ connected to the headsection 1. That is, by changing settable items and setting contentsdepending on whether the controller section is dedicated hardware or thepersonal computer, it is possible in either case to appropriatelyperform a setting regarding three-dimensional image processing. Further,a viewer program provided with a purpose of confirming an operation ofthe head section 1 and with a simple measurement function may beinstalled into the personal computer that functions as the controllersection 2′ so that an operation and a function of the connected headsection can be confirmed.

It is to be noted that the “distance image” obtained by using the imagecapturing part 10 and the light projecting part 20 shown in FIG. 1refers to an image in which a shade value of each pixel changes inaccordance with a distance from the image capturing part 10, whichcaptures the image of the workpiece WK, to the workpiece WK. In otherwords, the “distance image” can be said to be an image in which a shadevalue is decided based on the distance from the image capturing part 10to the workpiece WK. It can also be said to be a multi-level imagehaving a shade value in accordance with the distance to the workpieceWK. It can also be said to be a multi-level image having a shade valuein accordance with a height of the workpiece WK. Further, it also besaid to be a multi-level image obtained by converting the distance fromthe image capturing part 10 to a shade value with respect to each pixelof a brightness image.

As a technique for generating the distance image, there are roughlydivided two systems: one is a passive system (passive measurementsystem) for generating the distance image by use of an image captured onan illumination condition for obtaining a normal image; and the otherone is an active system (active measurement system) for generating thedistance image by actively performing irradiation with light formeasurement in a height direction. A representative technique of thepassive system is a stereo measurement method. In this technique, thedistance image can be generated only by preparing two or more imagecapturing parts 10 and disposing these two or more cameras in apredetermined positional relation, and hence it is possible to generatethe distance image through use of a general image processing system forgenerating a brightness image, so as to suppress system constructioncost.

On the other hand, representative techniques of the active system arethe light cutting method and the pattern projecting method. The lightcutting method is that in the foregoing stereo measurement method, theone camera is replaced with a light projecting part such as a lightprojector, linear laser light is projected to the workpiece, and thethree-dimensional shape of the workpiece is reproduced from a distortedcondition of an image of the linear light in accordance with a shape ofthe object surface. In the light cutting method, deciding thecorresponding point is unnecessary, thereby to allow stable measurement.However, since the measurement with respect to only one line is possibleper one measurement, when measured values of all the pixels are intendedto be obtained, the target or the camera needs scanning. As opposed tothis, the pattern projecting method is that a shape, a phase or the likeof a predetermined pattern projected to the workpiece is shifted tocapture a plurality of images and the captured plurality of images areanalyzed, to reproduce the three-dimensional shape of the workpiece.There are several sorts of pattern projecting methods, andrepresentative ones among them include: a phase shift method in which aphase of a sinusoidal wave fringe pattern is shifted to capture aplurality of (at least three or more) images and a phase of a sinusoidalwave with respect to each pixel from the plurality of images is found,to find three-dimensional coordinates on the surface of the workpiecethrough use of the found phase; a moire topography method in which thethree-dimensional shape is reproduced through use of a sort of wavingphenomenon of a spatial frequency when two regular patterns aresynthesized; a spatial coding method in which a pattern to be projectedto the workpiece is itself made different in each image-capturing, forexample, each of fringe patterns, whose fringe width with a monochromeduty ratio of 50% gets thinner to one-half, to one-quarter, toone-eighth, . . . of a screen, is sequentially projected, and a patternprojected image with each pattern is shot, to find an absolute phase ofthe height of the workpiece; and a multi-slit method in which apatterned illumination of a plurality of thin lines (multi-slit) isprojected to the workpiece and the pattern is moved at a pitch narrowerthan a slit cycle, to perform a plurality of times of shooting.

In the three-dimensional image processing apparatus 100 according to thepresent embodiment, the distance image is generated by the phase shiftmethod and the spatial coding method described above. This allowsgeneration of the distance image without relatively moving the workpieceor the head.

A disposition layout of the image capturing part 10 and the lightprojecting part 20 in FIG. 1 is made so as to hold the light projectingpart 20 obliquely and the image capturing part 10 vertically such thatlight is projected to the workpiece WK from an oblique direction andreflected light from the workpiece WK is received in almost a verticaldirection. By making the light projecting direction and the imagecapturing direction disagree and inclined with each other, it ispossible to capture a pattern projected image that has captured a shadecaused by the unevenness of the surface shape of the workpiece WK.

Second Embodiment

However, the present invention is not restricted to this dispositionexample, and for example, as in a three-dimensional image processingapparatus 200 according to a second embodiment shown in FIG. 3, adisposition example may be adopted where the image capturing part 10side is held obliquely with respect to the workpiece WK and the lightprojecting part 20 side is held vertically. Also by a head section 1B asthus disposed, similarly, it is possible to incline the light projectingdirection and the image capturing direction with each other, so as tocapture a pattern projected image that has captured the shade of theworkpiece WK.

Third Embodiment

Further, one of or both the light projecting part and the imagecapturing part can be disposed in a plurality of number. For example, asin a three-dimensional image processing apparatus 300 shown in FIG. 4Aas a third embodiment, while the image capturing part 10 is heldvertically with respect to the workpiece WK, two light projecting parts20 are disposed on both sides with the image capturing part 10 placed atthe center, which can thus be configured as a head section 1C thatprojects light from right and left. As thus described, by capturingpattern projected images while projecting light in different directions,it is possible to reduce a situation where the height measurement isinaccurate or impossible due to occurrence of the state of being unableto capture some part of the pattern projected image, such as the stateof the workpiece WK itself hiding a shade pattern by light projectionfrom one direction. Especially when the light projecting part 20 isdisposed so as to receive light projection from directions opposed toeach other with respect to the workpiece (e.g., right and leftdirections or front and rear directions), it is possible tosignificantly reduce the possibility that the image cannot be captureddue to being blocked by the workpiece itself.

Fourth Embodiment

The configuration has been described in the above example where the oneimage capturing part and the two light projecting parts are used, but incontrast, there can also be formed a configuration where two imagecapturing parts and one light projecting part are used. FIG. 4B showssuch an example as a three-dimensional image processing apparatus 400according to a fourth embodiment. In a head section 1D shown in thisexample, the light projecting part 20 is held vertically with respect tothe workpiece WK, and the image capturing parts 10 are disposed on theright and left to the light projecting part 20 in the drawing obliquelywith respect to the workpiece WK. Also in this configuration, it ispossible to capture images of the workpiece WK in different angles ofinclination, so as to suppress a situation where some part of thepattern projected image becomes difficult to capture, as in the thirdembodiment. Moreover, it is possible by this method to simultaneouslycapture two pattern projected images by one light projection, so as toalso obtain an advantage of being able to reduce the processing time.

On the other hand, there are differences in image-captured region, fieldof view, and the like between images of the same workpiece captured bythe two image capturing parts from different angles, and hence it isnecessary to perform an operation for making correspondence betweenpositions of pixels of one image and those of another, which can lead tooccurrence of an error. As opposed to this, according to the foregoingthird embodiment, with the image capturing part made common, images withthe same field of view can be captured when measurement light isprojected from either light projecting part, thereby eliminating theneed for such an integration operation as above, and it is possible toobtain an advantage of being able to avoid occurrence of an errorassociated with the integration operation, so as to simplify theprocessing.

It is to be noted that, although the embodiments have been describedabove where the image capturing part 10 and the light projecting part 20are integrally configured in each head section, the present invention isnot restricted to this configuration. For example, the head section canbe one where the image capturing part 10 and the light projecting part20 are made up of separate members. Further, it is also possible toprovide the image capturing parts and the light projecting parts innumber of three or more.

(Block Diagram)

Next, FIG. 5 shows a block diagram showing the configuration of thethree-dimensional image processing apparatus 300 according to the thirdembodiment of the present invention. As shown in FIG. 5, thethree-dimensional image processing apparatus 300 is provided with thehead section 1 and the controller section 2.

(Head Section 1)

This head section 1 is provided with the light projecting part 20, theimage capturing part 10, a head-side control section 30, a head-sidecomputing section 31, a head-side storage part 38, the head-sidecommunication part 36, and the like. The light projecting part 20includes a measurement light source 21, a pattern generating part 22 anda plurality of lenses 23, 24, 25. The image capturing part 10 includes acamera and a plurality of lenses, though not shown.

(Light Projecting Part 20)

The light projecting part 20 is a member for projecting incident lightas structured illumination of a predetermined projection pattern fromthe oblique direction with respect to an optical axis of the imagecapturing part. A projector can be used as this light projecting part20, and it includes a lens as an optical member, the pattern generatingpart 22, and the like. The light projecting part 20 is disposedobliquely above the position of the workpiece that stops or moves. It isto be noted that the head section 1 can include a plurality of lightprojecting parts 20. In the example of FIG. 5, the head section 1includes two light projecting parts 20. Here, there are disposed a firstprojector 20A capable of irradiating the workpiece with measuringillumination light from a first direction (right side in FIG. 5) and asecond projector 20B capable of irradiating the workpiece with measuringillumination light from a second direction which is different from thefirst direction (left side in FIG. 5). The first projector 20A and thesecond projector 20B are disposed symmetrically, with the optical axisof the image capturing part 10 placed therebetween. Measurement light isprojected to the workpiece alternately from the first projector 20A andthe second projector 20B, and pattern images of the respective reflectedlight are captured in the image capturing part 10.

As the measurement light source 21 of each of the first projector 20Aand the second projector 20B, for example, a halogen lamp that emitswhite light, a white LED (light emitting diode) that emits white light,or the like can be used. Measurement light emitted from the measurementlight source 21 is appropriately collected by the lens, and is thenincident on the pattern generating part 22.

Further, on top of the light projecting part that emits the measurementlight for acquiring the pattern projected image to generate the distanceimage, an observing illumination light source for capturing a normaloptical image (brightness image) can also be provided. As the observingillumination light source, in addition to the LED, a semiconductor laser(LD), a halogen lamp, an HID (High Intensity Discharge), or the like canbe used. Especially in the case of using an element capable of capturinga color image as the image capturing element, a white light source canbe used as the observing illumination light source.

Measurement light emitted from the measurement light source 21 isappropriately collected by the lens 23, and is then incident on thepattern generating part 22. The pattern generating part 22 can realizeillumination of an arbitrary pattern. For example, it can invert thepattern in accordance with colors of the workpiece and the background,such as black on a white background or white on a black background, soas to express an appropriate pattern easy to see or easy to measure. Assuch a pattern generating part 22, for example, a DMD (DigitalMicro-mirror Device) can be used. FIG. 6 shows an example of the DMD. ADMD 39 shown in this drawing can express an arbitrary pattern byswitching on/off a minute mirror (micro-mirror MC) with respect to eachpixel. This allows easy irradiation with the pattern with black andwhite inverted. Using the DMD as the pattern generating part 22 allowseasy generation of the arbitrary pattern and eliminates the need forpreparing a mechanical pattern mask and performing an operation forreplacing the mask, thus leading to an advantage of being able to reducethe size of the apparatus and perform rapid measurement. Further, sincethe pattern generating part 22 configured with the DMD can be used in asimilar manner to normal illumination by performing irradiation with afull-illumination pattern being turned on for all the pixels, it canalso be used for capturing the brightness image.

Each micro-mirror MC is formed in a rectangular shape. In the example ofthe DMD 39 of FIG. 6, the micro-mirrors MC are each arranged in aposture inclined in a diamond shape. As a result, it has an advantage ofbeing able to neatly draw each line constituting a projection pattern atthe time of forming the projection pattern in the shape of diagonalfringes as shown in FIG. 7. In contrast, in an example of a DMD 39′where micro-mirrors MC′ are arranged in a grid pattern as shown in FIG.8, each line becomes jagged at the time of drawing a diagonal projectionpattern. For this reason, in the case of projecting a projection patterninclined diagonally as described later, it is possible to reduce theroughness of the fringes by use of the inclined arrangement type DMD asin FIG. 7.

It is to be noted that the pattern generating part 22 is not restrictedto the DMD, but can also be an LCD (Liquid Crystal Display), an LCOS(Liquid Crystal on Silicon: reflective liquid crystal element), or amask. The measurement light having been incident on the patterngenerating part 22 is converted to light with a previously set patternand a previously set intensity (brightness), and then emitted. Themeasurement light emitted from the pattern generating part 22 isconverted to light having a larger diameter than an observable andmeasurable field of view of the image capturing part 10 by means of theplurality of lenses, and thereafter the workpiece is irradiated with theconverted light.

(Image Capturing Part 10)

The image capturing part 10 is provided with a camera for acquiringreflected light that is projected by the light projecting part 20 andreflected on the workpiece WK, to capture a plurality of patternprojected images. As such a camera, a CCD, a CMOS or the like can beused. In this example, there is used a monochrome CCD camera that canobtain a high resolution. In addition, it goes without saying that acamera capable of capturing a color image can also be used. Further, theimage capturing part can also capture a normal brightness image inaddition to the pattern projected image.

The head-side control section 30 is a member for controlling the imagecapturing part 10, as well as the first projector 20A and the secondprojector 20B which are the light projecting part 20. The head-sidecontrol section 30, for example, creates a light projection pattern forthe light projecting part 20 projecting the measurement light to theworkpiece to obtain the pattern projected image. The head-side controlsection 30 makes the image capturing part 10 capture a phase shift imagewhile making the light projecting part 20 project a projection patternfor phase shifting, and further, makes the image capturing part 10capture a spatial code image while making the light projecting part 20project a projection pattern for spatial coding. In such a manner, thehead-side control section 30 functions as a light projection controllingpart for controlling the light projecting part such that the phase shiftimage and the spatial code image can be captured in the image capturingpart 10.

The head-side computing section 31 includes a filter processing section34 and a distance image generating part 32. The distance imagegenerating part 32 generates the distance image based on the pluralityof pattern projected images captured in the image capturing part 10.

A head-side storage part 38 is a member for holding a variety ofsettings, images and the like, and a storage element such as asemiconductor memory or a hard disk can be used. For example, itincludes a brightness image storage section 38 b for holding the patternprojected image captured in the image capturing part 10, and a distanceimage storage section 38 a for holding the distance image generated inthe distance image generating part 32.

The head-side communication part 36 is a member for communicating withthe controller section 2. Here, it is connected with a controller-sidecommunication part 42 of the controller section 2. For example, thedistance image generated in the distance image generating part 32 istransmitted to the controller section 2.

(Distance Image Generating Part 32)

The distance image generating part 32 is a part for generating thedistance image where the shade value of each pixel changes in accordancewith the distance from the image capturing part 10, which captures theimage of the workpiece WK, to the workpiece WK. For example, in the caseof generating the distance image by the phase shift method, thehead-side control section 30 controls the light projecting part 20 so asto project a sinusoidal fringe pattern to the workpiece while shiftingits phase, and the head-side control section 30 controls the imagecapturing part 10 so as to capture a plurality of images with the phaseof the sinusoidal fringe pattern shifted in accordance with the aboveshifting. Then, the head-side control section 30 finds a sinusoidalphase with respect to each pixel from the plurality of images, togenerate the distance image through use of the found phases.

Further, in the case of generating the distance image by use of thespatial coding method, a space that is irradiated with light is dividedinto a large number of small spaces each having a substantially fan-likecross section, and these small spaces are provided with a series ofspatial code numbers. For this reason, even when the height of theworkpiece is large, in other words, even when the height difference islarge, the height can be computed from the spatial code numbers so longas the workpiece is within the space irradiated with light. Hence it ispossible to measure the whole shape of even the workpiece having a largeheight. Combining the phase shift method and the spatial coding methodallows accurate measurement of a three-dimensional shape from a lowportion to a high portion.

Generating the distance image on the head section side and transmittingit to the controller section in such a manner can lead to reduction inamount of data to be transmitted from the head section to the controllersection, thereby avoiding a delay in the processing which can occur dueto transmission of a large amount of data.

It is to be noted that, although the distance image generatingprocessing is to be performed on the head section 1 side in the presentembodiment, for example, the distance image generating processing can beperformed on the controller section 2 side. Further, the tone conversionfrom the distance image to the low-tone distance image can be performednot only in the controller section but also on the head section side. Inthis case, the head-side computing section 31 realizes a function of thetone conversion part.

(Controller Section 2)

Further, the controller section 2 is provided with the controller-sidecommunication part 42, a controller-side control section, acontroller-side computing section, a controller-side storage part, aninspection executing part 50, and a controller-side setting part 41. Thecontroller-side communication part 42 is connected with the head-sidecommunication part 36 of the head section 1 and performs datacommunication. The controller-side control section is a member forcontrolling each member. The controller-side computing section realizesa function of an image processing section 60.

The image processing section 60 realizes functions of an image searchingpart 64, the tone converting part 46, and the like.

(Tone Converting Part)

Based on the distance image, the tone converting part 46 tone-convertsthe high-tone distance image to the low-tone distance image (itsprocedure will be detailed later). Herewith, the distance image havingthe height information generated in the head section is expressed as alow-tone shade image that can also be handled by existing imageprocessing, and this can contribute to the measurement processing andthe inspection processing. Further, there can also be obtained anadvantage of being able to disperse a load by making the distance imagegenerating processing and the tone conversion processing shared by thehead section and the controller section. It is to be noted that inaddition to the distance image, the low-tone distance image may also begenerated on the head section side. Such processing can be performed inthe head-side computing section. This can further alleviate the load onthe controller section side, to allow an efficient operation.

Further, the tone converting part does not tone-convert the whole of thedistance image, but preferably selects only a necessary portion thereofand tone-converts it. Specifically, it tone-converts only a portioncorresponding to an inspection target region previously set by aninspection target region setting part (detailed later). In such amanner, the processing for converting the multi-tone distance image tothe low-tone distance image is restricted only to the inspection targetregion, thereby allowing alleviation of the load necessary for the toneconversion. Moreover, this also contributes to reduction in processingtime. That is, improving the reduction in processing time can lead topreferable use in an application with limited processing time, such asan inspection in a FA application, thereby to realize real-timeprocessing.

The controller-side storage part is a member for holding a variety ofsettings and images, and the semiconductor storage element, the harddisk, or the like can be used.

The controller-side setting part 41 is a member for performing a varietyof settings to the controller section, and accepts an operation from theuser via the input part 3 such as a console connected to the controllersection, to instruct a necessary condition and the like to thecontroller side. For example, it realizes functions of a tone conversioncondition setting part 43, a reference plane specifying part 44, aspatial coding switching part 45, an interval equalization processingsetting part 47, a light projection switching part 48, a shutter speedsetting part 49, and the like.

The inspection executing part 50 executes predetermined inspectionprocessing on the low-tone distance image tone-converted in the toneconverting part 46.

(Hardware Configuration)

Next, a constitutional example of hardware of the controller section 2is shown in a block diagram of FIG. 9. The controller section 2 shown inthis drawing has a main control section 51 for performing control ofeach section of the hardware while performing numerical valuecalculation and information processing based on a variety of programs.The main control section 51, for example, has a CPU as a centralprocessing part, a working memory such as a RAM that functions as aworking area for the main control section 51 at the time of executing avariety of programs, a program memory such as a ROM, a flash ROM or anEEPROM where a start-up program, an initialization program and the likeare stored.

Further, the controller section 2 is provided with: a controller-sideconnection section 52 for connecting with the head section 1 thatincludes the image capturing part 10, the light projecting part 20 andthe like, controlling the light projecting part 20 so as to projectlight with a sinusoidal fringe pattern to the workpiece while shiftingits phase, and fetching image data obtained by the image capturing inthe image capturing part 10; an operation inputting section 53 which isinputted with an operation signal from the input part 3; a displaycontrolling section 54 configured of a display DSP that allows thedisplay part 4, such as the liquid crystal panel, to display an image,and the like; a communication section 55 communicably connected to theexternal PLC 70, the personal computer PC and the like; a RAM 56 forholding temporary data; a controller-side storage part 57 for storing asetting content; an auxiliary storage part 58 for holding data set bymeans of the three-dimensional image processing program installed in thepersonal computer PC; an image processing section 60 configured of acomputing DSP that executes the measurement processing such as the edgedetection and the area calculation, and the like; an output section 59for outputting a result of performing a predetermined inspection basedon a result of the processing in the image processing section 60, or thelike; and some other section. Each of the above hardware is communicablyconnected via an electric communication path (wiring) such as a bus.

In the program memory in the main control section 51, there is stored acontrol program for controlling each of the controller-side connectionsection 52, the operation inputting section 53, the display controllingsection 54, the communication section 55 and the image processingsection 60 by a command of the CPU, or the like. Further, the foregoingprocessing sequence program, namely the processing sequence programgenerated in the personal computer PC and transmitted from the personalcomputer PC, is stored into the program memory.

The communication section 55 functions as an interface (I/F) thatreceives an image capturing trigger signal from the PLC 70 at the timewhen a trigger is inputted in a sensor (photoelectronic sensor, etc.)connected to the external PLC 70. Further, it also functions as aninterface (I/F) that receives the three-dimensional image processingprogram transmitted from the personal computer PC, layout informationthat prescribes a display mode of the display part 4, and the like.

When the CPU of the main control section 51 receives the image capturingtrigger signal from the PLC 70 via the communication section 55, ittransmits an image capturing command to the controller-side connectionsection 52. Further, based on the processing sequence program, ittransmits to the image processing section 60 a command to instruct imageprocessing to be executed. It should be noted that such a configurationcan be formed where, as the apparatus for generating the image capturingtrigger signal, not the PLC 70 but a trigger inputting sensor such as aphotoelectronic sensor may be directly connected to the communicationsection 55.

The operation inputting section 53 functions as an interface (I/F) forreceiving an operation signal from the input part 3 based on a user'soperation. A content of the user's operation by use of the input part 3is displayed on the display part 4. For example, in the case of usingthe console as the input part 3, each component such as a cross key forvertically and horizontally moving a cursor that is displayed on thedisplay part 4, a decision button or a cancel button can be disposed. Byoperating each of these components, the user can, on the display part 4,create a flowchart that prescribes a processing sequence for imageprocessing, edit a parameter value of each image processing, set areference region, and edit a reference registered image.

The controller-side connection section 52 fetches image data.Specifically, for example, when receiving the image capturing commandfor the image capturing part 10 from the CPU, the controller-sideconnection section 52 transmits an image data fetching signal to theimage capturing part 10. Then, after image capturing has been performedin the image capturing part 10, it fetches image data obtained by theimage capturing. The fetched image data is once stored in a buffer(cache), and substituted in a previously prepared image variable. Itshould be noted that, differently from a normal variable dealing with anumerical value, the “image variable” refers to a variable allocated asan input image of a corresponding image processing unit, to be set as areference destination of measurement processing or image display.

The image processing section 60 executes the measurement processing onthe image data. Specifically, first, the controller-side connectionsection 52 reads the image data from a frame buffer while referring tothe foregoing image variable, and internally transmits it to a memory inthe image processing section 60. Then, the image processing section 60reads the image data stored in the memory and executes the measurementprocessing. Further, the image processing section 60 includes the toneconverting part 46, an abnormal point highlight part 62, the imagesearching part 64, and the like.

Based on a display command transmitted from the CPU, the displaycontrolling section 54 transmits to the display part 4 a control signalfor displaying a predetermined image (video). For example, it transmitsthe control signal to the display part 4 in order to display image databefore or after the measurement processing. Further, the displaycontrolling section 54 also transmits a control signal for allowing thecontent of the user's operation by use of the input part 3 to bedisplayed on the display part 4.

The head section 1 and the controller section 2 made up of such hardwareas above are configured to be able to realize each part or function ofFIG. 5 by way of a variety of programs in forms of software. In thisexample, there is adopted the mode of installing the three-dimensionalimage processing program into the computer of FIG. 1 to perform asetting necessary for the three-dimensional image processing.

(Tone Conversion)

The above three-dimensional image processing apparatus acquires adistance image of the workpiece, performs image processing on thisdistance image, and inspects its result. The three-dimensional imageprocessing apparatus according to the present embodiment can execute twosorts of inspections: image inspection processing for performingcomputing by use of information of an area, an edge or the like by meansof existing hardware, on top of height inspection processing forperforming computing by use of height information as it is as a pixelvalue of the distance image. Here, for sustaining the accuracy in heightinspection processing, it is necessary to generate a multi-tone distanceimage. On the other hand, the image inspection processing cannot beexecuted on such a multi-tone distance image by means of the existinghardware. Therefore, in order to perform the image inspection processingby use of the existing hardware, the multi-tone distance image issubjected to the tone conversion, to generate a low-tone distance image.

However, when height information of the multi-tone distance image isconverted as it is to the low-tone distance image, the accuracy of theheight information is lost, which is problematic. Many of general imagesused in the FA application or the like are images each expressing ashade value by 8 tones in monochrome. As opposed to this, the high-toneimage such as a 16-tone image is used as the distance image. For thisreason, at the time of tone-converting the multi-tone distance image tothe low-tone distance image, a considerable amount of height informationis lost, which affects the accuracy in inspection. Having said that,increasing the number of tones of the image handled in the existingimage processing for enhancing the accuracy results in a steep rise ofintroduction cost and an increased processing load, making its use moredifficult.

Accordingly, at the time of the tone conversion as thus described, it isnecessary to set such a condition for the tone conversion as to sustainnecessary height information. Hereinafter, a method and a sequencetherefor will be described in detail.

(Height Inspection or Image Inspection)

First, a description will be given of a processing operation ofperforming the height inspection processing by use of thethree-dimensional image processing apparatus based on a flowchart ofFIG. 10. This three-dimensional image processing apparatus is providedwith, as tools for performing calculation processing, a heightinspection processing tool for performing the height inspection on thedistance image, and a variety of image inspection processing tools forperforming the image inspection on the existing brightness image. Here,the height inspection processing will be described.

First, a distance image is generated (Step S71). Specifically, thedistance image generating part 32 generates the distance image by use ofthe image capturing part 10 and the light projecting part 20.Subsequently, desired calculation processing is selected (Step S72).Here, a tool necessary for the calculation processing is selected.

In the case of selecting the inspection processing tool, the processinggoes to Step S73, and the tone conversion processing is performed on thehigh-tone distance image obtained in Step S71 above, to convert it to alow-tone distance image. This makes it possible for even an inspectionprocessing tool in the existing image processing apparatus to handle alow-tone distance image. It is to be noted that the tone conversionprocessing is not performed in the whole region of the high-tonedistance image, but is preferably performed only within an inspectiontarget region having been set for the image inspection processing.

Meanwhile, in the case of selecting the height inspection tool, sinceheight information in the multi-tone distance image is used as it is,the processing goes to Step S74 without performing the tone conversion.

Further, the inspection executing part 50 performs a variety ofcalculation processing (Step S74), and then determines whether or notthe workpiece is a non-defective product based on a result of thecalculation (Step S75). When it is determined by the inspectionexecuting part 50 that the workpiece is a non-defective product (StepS75: YES), a determination signal outputting part 160 outputs an OKsignal as a determination signal to the PLC 70 (Step S76), and when itis determined by the inspection executing part 50 that the workpiece isnot a non-defective product, namely a defective product (Step S75: NO),the determination signal outputting part 160 outputs an NG signal as adetermination signal to the PLC 70 (Step S77).

(Incorrect-Height Determining Function)

The three-dimensional image processing apparatus is further providedwith an incorrect-height determining function of detecting that anincorrect value different from original height information of theworkpiece is detected due to multi-reflection when the workpiece isirradiated with a projection pattern and a pattern projected image iscaptured. A block diagram of FIG. 11 shows an example of such athree-dimensional image processing apparatus. The three-dimensionalimage processing apparatus shown in this drawing is provided with thelight projecting parts 20, the image capturing part 10, a head-sidecontrol section for controlling the light projecting parts 20 and theimage capturing part 10, and a head-side computing section. Thehead-side computing section is provided with the distance imagegenerating part 32 and an incorrect-height determining part 37. Bycombining the phase shift method and the spatial coding method, thedistance image generating part 32 can obtain a distance image thatincludes height information with high accuracy in a relatively shortperiod of time. The pattern generating part 22 generates a firstprojection pattern and a second projection pattern whose fringedirection is different from that of the first projection pattern as aplurality of projection patterns obtained by changing a fringe directionof the projection pattern. The incorrect-height determining part 37makes comparison in height information of a corresponding portion of theworkpiece WK between a first distance image, generated in the distanceimage generating part 32 based on a first pattern projected imageobtained by projecting from the light projecting parts 20 the firstprojection pattern generated in the pattern generating part 22 andcapturing an image thereof in the image capturing part 10, and a seconddistance image, generated based on a second pattern projected imageobtained by the second projection pattern projected from the commonlight projecting parts 20, to determine height information of a portionwhere a difference not smaller than a predetermined value has occurredas incorrect.

In such a manner, by projecting different projection patterns from thesame light projecting part 20 and computing pieces of height informationthereof, the same height information is obtained from either projectionpattern in the case of a normal result, whereas different pieces ofheight information are obtained depending on the projection pattern whenmulti-reflection, such as mirror reflection, has occurred. Accordingly,by comparing pieces of height information computed with the respectiveprojection patterns, in the case of existence of a portion where aremarkable difference is seen, it can be determined that themulti-reflection has occurred and correct height information has notbeen obtained in the portion. In such a manner, a portion where anincorrect measurement result has occurred due to the mirror reflectionor the like can be detected, which has hitherto been difficult, and theportion where it has occurred can further be specified, thereby to allowimprovement in reliability of a measurement result.

(Pattern Generating Part 22)

The first projection pattern and the second projection pattern areprojected from the same light projecting part 20, namely a light source,to capture projection pattern images. It is to be noted that for thepurpose of reducing computational processing, two kinds of projectionpatterns are preferably projected, but it is also possible to employ aconfiguration where three or more kinds of projection patterns areprojected and pieces of height information obtained from the respectiveprojection pattern images are compared. However, when the number ofprojection pattern is large, it takes a long time to perform irradiationwith the projection patterns, capture projection pattern images andcompute height information.

Here, as examples of the projection pattern generated by the patterngenerating part 22, FIG. 12 shows a first projection pattern PN1 andFIG. 13 shows a second projection pattern PN2. In these examples, thefirst projection pattern PN1 is inclined by +45° with respect to avertical direction as compared with a conventional projection patternPN0 shown in FIG. 24. On the other hand, the second projection patternPN2 is inclined by −45°. For drawing the projection patterns as thusinclined, a DMD arranged with micro-mirrors in a diamond shape as inFIGS. 6 and 7 described above can be preferably used. In addition, asfor an angle of inclination of each of the first projection pattern andthe second projection pattern, it is enough if each pattern is inclined.Conversely, each pattern functions unless being a vertical or horizontalpattern, and the angle of inclination thereof does not matter. However,in the case of the patterns inclined by ±45° as described above, therecan be obtained an advantage of facilitating constituting a neat fringepattern by means of the DMD arranged with micro-mirrors each formed in adiamond shape. Further, the first projection pattern and the secondprojection pattern are preferably linearly symmetrical patterns.

Next, based on a flowchart of FIG. 14, a description will be given belowof a procedure for capturing images of the workpieces WKA, WKB and WKCas shown in FIGS. 26 and 27 by use of the projection patterns of FIGS.12 and 13, to perform incorrect-height determination. First, in StepS1401, the workpieces are irradiated with a plurality of patterns by thephase shift method and the spatial coding method along a direction ofthe first projection pattern, to capture a first projection patternimage group. Here, as shown in FIG. 26, the workpieces WKA, WKB and WKCare irradiated with the first projection pattern from the lightprojecting part 20, to capture a first projection pattern image in theimage capturing part 10. FIGS. 15A and 15B show an example of theobtained first projection pattern image. In these drawings, it can beconfirmed that the mirror reflection has occurred on the side surfacesof the workpiece WKB and the workpiece WKC as indicated by arrows in anenlarged view of FIG. 15B.

Next, in Step S1402, the workpieces are irradiated with a plurality ofpatterns from the same light projecting part 20 by the phase shiftmethod and the spatial coding method along a direction of the secondprojection pattern this time, to capture a second projection patternimage group. FIGS. 16A and 16B show an example of the obtained secondprojection pattern image. Also here, it can be confirmed that the mirrorreflection has occurred on the side surfaces of the workpiece WKB andthe workpiece WKC as indicated in an enlarged view of FIG. 16B. Itshould be noted that, when compared with the first projection patternimages of FIGS. 15A and 15B, the fringe pattern is naturally differenttherebetween.

Further in Step S1403, a first distance image and a second distanceimage each including height information are generated in the distanceimage generating part 32 from the first projection pattern image groupand the second projection pattern image group, respectively. As aresult, height information of each portion of the workpieces isobtained. In addition, the first distance image and the second distanceimage are generated after the first projection pattern image group andthe second projection pattern image group have been captured in thisexample, but it is also possible to first capture the first projectionpattern image group and generate the first distance image, andthereafter capture the second projection pattern image group andgenerate the second distance image.

FIGS. 17A, 17B and 17C show the obtained pieces of height information asimages of the workpieces. In these drawings, FIG. 17A shows originalheights of the workpieces, FIG. 17B shows heights obtained from thefirst projection pattern image, and FIG. 17C shows heights obtained fromthe second projection pattern image. As apparent by comprising these, itis found that incorrect heights have been detected on the side surfaceof the workpiece WKB and the side surface of the workpiece WKC, andnormal height information has been detected in the other portion.Specifically, according to the height information computed from thefirst projection pattern image shown in FIG. 17B, a result close to theactual height shown in FIG. 17A has been obtained on the side surface ofthe workpiece WKB, but such an incorrect result has been obtained on theside surface of the workpiece WKC that the side surface is continuouswith the stage ST at the same height where the workpieces are placed. Onthe other hand, according to the height information computed from thesecond projection pattern image shown in FIG. 17C, the side surface ofthe workpiece WKB and the side surface of the workpiece WKC both showincorrect results that the side surfaces further go below the stage ST.

The reason why the different results have occurred in the projectionpattern images, taken from the same direction (light source) withrespect to the same workpiece, as an outcome of the mirror reflection isthat a difference in direction of the projection pattern causesoccurrence of a difference in change of the projection pattern to beobserved, thus leading to a difference in computing of a distance bytriangulation, as described hereinafter. This situation will besequentially described based on FIGS. 18A to 20C. First, FIGS. 18A to18C show a situation where irradiation with the first projection patterninclined by +45° is performed to obtain height information of the sidesurface of the workpiece WKB. When the side surface of the workpiece WKBis irradiated with the first projection pattern and an image is capturedin the image capturing part 10 as in FIG. 18C, a projection patternimage as shown in FIG. 18A should originally be obtained. However,practically, a projection pattern image as shown in FIG. 18B is captureddue to the mirror reflection. When a focus is placed on heightinformation at a measurement point P3, it should originally beirradiated with a fringe 2, and hence in FIG. 18C, a height at anintersection between the fringe 2 and a visual line to the observationpoint extending from the image capturing part 10 is obtained.Practically, as shown in FIG. 18B, the irradiation pattern with whichthe stage ST is irradiated has been copied as if on a mirror due to themirror reflection, but since this is a pattern close to the originalpattern of FIG. 18A, a difference in obtained height is small, and alarge error has not occurred.

On the other hand, FIGS. 19A to 19C show a situation where the sidesurface of the workpiece WKB is irradiated with a different projectionpattern, here the second projection pattern inclined by −45°, to obtainheight information. In this state, as a result of the mirror reflection,a projection pattern image greatly different from a projection patternimage that should originally be obtained as shown in FIG. 19A has beencaptured as in FIG. 19B. Consequently, height information at anobservation point P₄ should originally be computed by means of thefringe 2 as shown in FIG. 19A, but practically, it has been computed bymeans of the next fringe 1 as shown in FIG. 19B. As a result, as shownin FIG. 19C, it has been computed as a height smaller than the stage ST,differently from the original height. A cause of occurrence of such anerror in the measurement result of the height is as follows: in thepattern projecting methods such as the phase shift method and thespatial coding method, a predetermined number of times of patternprojection are performed while a phase is shifted in each projectionpattern, but a phase of each pixel computed from the obtained pluralityof projection pattern images is displaced due to the mirror reflection,leading to occurrence of a difference in the measurement result.

Further, the height of the side surface of the workpiece WKC is alsoconsidered. First, FIGS. 20A to 20C show a situation where heightinformation of the side surface of the workpiece WKC irradiated with thefirst projection pattern is obtained. Here, as a result of the mirrorreflection, there has been captured a projection pattern image as inFIG. 20B, different from a projection pattern image that shouldoriginally be obtained as shown in FIG. 20A. Consequently, heightinformation at an observation point P₅ should originally be computed bymeans of a fringe 6 as shown in FIG. 20A, but practically, it has beencomputed by means of the fringe 5 as shown in FIG. 20B. As a result, asshown in FIG. 20C, it has been computed as a height smaller than anintersection between the fringe 6 as the original height and a visualline of the observation point (an intersection with the fringe 5 has thesame height as that of the stage).

On the other hand, FIGS. 21A to 21C show a situation where heightinformation of the side surface of the workpiece WKC irradiated with thesecond projection pattern is obtained. Also here, as a result of themirror reflection, there has been captured a projection pattern image asin FIG. 21B, different from a projection pattern image shown in FIG. 21Awhich should originally be obtained. Consequently, height information atan observation point P₆ should originally be computed by means of thefringe 5 as shown in FIG. 21A, but practically, it has been computed bymeans of the fringe 4 as shown in FIG. 21B. As a result, as shown inFIG. 21C, it has been computed as a height smaller than an intersectionbetween the fringe 5 as the original height and a visual line of theobservation point (an intersection with the fringe 4 has a smallerheight than that of the stage).

As thus described, how the effect due to the mirror reflection appearsvaries depending on the projection pattern, and hence there occurs astate where a computing result of height information is different onlyin the portion where the mirror reflection has occurred. Conversely,when the projection pattern is changed, the occurrence of themulti-reflection can be detected by use of the character of being ableto obtain different height information in a portion where themulti-reflection has occurred. That is, height information of the firstdistance image and that of the second distance image are compared for acorresponding portion, and height information of a portion where adifference not smaller than a predetermined threshold has occurred isdetermined as incorrect by the incorrect-height determining part 37.Specifically, returning to FIG. 14 for description, first in Step S1404,the position is moved to the initial positions. In Step S1405, heightinformation of the first distance image and that of the second distanceimage in the moved position are extracted and compared to each other,and the incorrect-height determining part 37 determines whether or not adifference therebetween is not smaller than the predetermined threshold.When a difference not smaller than the threshold is detected, theprocessing goes to Step S1406, to output the incorrectness. Here, a flagindicating the incorrectness is written in the pixel, and the processinggoes to Step S1407. On the other hand, when a difference is smaller thanthe threshold, the processing goes to Step S1407 while Step S1406 isskipped, and it is determined whether the determination processing hascompleted in all positions. When it has not, the processing goes to StepS1408 to move the position, namely the pixel, and thereafter theprocessing returns to Step S1405, to repeat the processing. Then, whenthe determination processing on all the pixels has completed, theprocessing is completed.

It is to be noted that in this example, height information are comparedwith respect to each of all pixels, but this example is not restrictive,and the processing can be simplified by making the comparison by unitsof several pixels. Further, the processing for comparing heightinformation can be simplified by such means as allowing the user topreviously specify a region where pieces of height information arecompared, specifically a region where the workpiece exists, orautomatically setting a target region by image processing.

In such a manner, it is possible to remove incorrect height informationby performing simple computing while using not any additional member butthe existing equipment, so as to obtain an advantage of being able toperform measurement and determination only by means of three-dimensionalinformation having high reliability.

In addition, when detecting the incorrectness of height information, theincorrect-height determining part 37 records a flag indicating anincorrect value as height information of the portion. Herewith, at thetime of performing computing by use of the distance image, it ispossible to perform processing not affected by an incorrect result bynot computing a portion with the flag, or by some other means. Further,such a portion where height information is determined as incorrect maybe treated in a similar manner to, for example, a portion where a shadehas generated due to irradiation with a projection pattern, a portionwhere phase-shift contrast is low and measurement cannot be performed,or some other portion.

Moreover, when computing that includes an incorrect value is specified,processing can be advanced in accordance with a use or a purpose in sucha manner that, by notifying the user that height information isincorrect or by some other means, the user is promoted to performresetting, or the computing is suspended, or is performed on the premisethat the information is incorrect. It is thus possible to avoid asituation where computing or the like is performed without being awarethat the information has low reliability, so as to hold credibility of ameasurement result.

Meanwhile, with two distance images generated, by averaging twomeasurement results in a portion determined as not incorrect, it ispossible to seek higher accuracy in height information. For example, thefirst distance image as shown in FIG. 22A is compared with the seconddistance image shown in FIG. 22B, to extract and remove each incorrectplace EA where a difference not smaller than a threshold has occurred inheight information as in FIG. 22C, and the remaining portion isaveraged, to generate a synthetic height image as shown in FIG. 22D.This can avoid mixture of a correct measurement result and an incorrectmeasurement result, and suppress a harmful effect on the post-stageprocessing due to the incorrect result. Additionally, averaging the twodistance images can also suppress a measurement error, and furtherimprove the measurement accuracy of height information.

In the present example based on the third embodiment, two lightprojecting parts 20A and 20B are provided as shown in FIG. 4A, but inthe foregoing description, one (e.g., the light projecting part 20B thatprojects light from the left side) out of those light projecting partshas been used. A similar processing can also be performed using theother light projecting part (e.g. the light projecting part 20A thatprojects light from the right side). Herewith, it is possible togenerate two distance images, highly accurate average distance imageobtained by removing an incorrect place by one light projecting part,and a highly accurate averaged distance image obtained from a differentangle by the other light projecting part. (Internally, the first andsecond projection patterns are projected by each of the light projectingparts, and hence a total of four kinds of distance images aregenerated.) Since one measurement system can be constituted by one imagecapturing part and one light projecting part, in the embodiment wherethe two light projecting parts are provided, two measurement systems canbe constructed. By further synthesizing highly accurate averageddistance images obtained by these two different measurement systems, itis possible to generate a highly accurate distance image where a portionthat is shaded by light projection is compensated.

In addition, this synthesis may be performed in a sequence describedbelow. First, synthesis is performed between the respective measurementsystems by use of the first projection pattern, and thereafter,synthesis is performed between the respective measurement systems by useof the second projection pattern. Finally, the obtained two syntheticimages are compared, an incorrect place where a difference not smallerthan a threshold has occurred in height information is removed, followedby averaging of the remaining portion, to generate a highly accuratedistance image. Alternatively, it is also possible that synthesis isperformed with respect to each light projection system by use of thefirst projection pattern and the second projection pattern, thesynthetic images obtained by the respective light projection systems arethen compared, and similarly to the above, an incorrect place where adifference not smaller than a threshold has occurred in heightinformation is removed, followed by averaging of the remaining portion,to generate a highly accurate distance image. As thus described, evenwhen the procedure for synthesis is changed, there can be obtained asimilar effect that an incorrect place is removed and the remainingportion is made highly accurate.

A three-dimensional image processing apparatus, a three-dimensionalimage processing method, a three-dimensional image processing program, acomputer-readable recording medium, and a recording device according tothe present invention are applicable to an inspection apparatus usingthe principle of triangulation, and the like.

What is claimed is:
 1. A three-dimensional image processing apparatuswhich is capable of acquiring a distance image that includes heightinformation of an inspection target and also performing image processingbased on the distance image, the apparatus comprising: a lightprojecting part for projecting incident light as structured illuminationof a predetermined fringe-like projection pattern from an obliquedirection with respect to an optical axis of a below-described imagecapturing part; the image capturing part for acquiring reflected lightthat is projected by the light projecting part and reflected on aninspection target, to capture a plurality of pattern projected images; adistance image generating part capable of generating a distance imagebased on the plurality of images captured in the image capturing part; apattern generating part for generating a first projection pattern and asecond projection pattern whose fringe direction is different from thatof the first projection pattern as a plurality of projection patternsobtained by changing a fringe direction of the projection pattern; andan incorrect-height determining part for making comparison in heightinformation of a corresponding portion of the inspection target betweena first distance image, generated in the distance image generating partbased on a first pattern projected image obtained by projecting from thelight-projecting part the first projection pattern generated in thepattern generating part and capturing an image of the first projectionpattern in the image capturing part, and a second distance image,generated based on a second pattern projected image obtained by thesecond projection pattern projected from the common light projectingpart, to determine height information of a portion where a differencenot smaller than a predetermined value occurs as incorrect.
 2. Thethree-dimensional image processing apparatus according to claim 1,wherein the pattern generating part generates as the first projectionpattern a projection pattern whose fringe direction is inclined withrespect to a vertical direction and a horizontal direction.
 3. Thethree-dimensional image processing apparatus according to claim 2,wherein the pattern generating part generates as the first projectionpattern a projection pattern whose fringe direction is inclined by 45°from the vertical direction.
 4. The three-dimensional image processingapparatus according to claim 1, wherein the pattern generating partgenerates as the second projection pattern a projection pattern that issymmetrical to the first projection pattern.
 5. The three-dimensionalimage processing apparatus according to claim 1, wherein the patterngenerating part includes a digital micro-mirror device.
 6. Thethree-dimensional image processing apparatus according to claim 5,wherein a micro-mirror constituting each pixel of the digitalmicro-mirror device has a rectangular shape in a plan view, and eachmicro-mirror is arranged in a posture inclined in a diamond shape. 7.The three-dimensional image processing apparatus according to claim 1,wherein the distance image generating part is capable of generating asynthetic image obtained by averaging the first distance image and thesecond distance image except for a portion determined as incorrect inthe incorrect-height determining part.
 8. The three-dimensional imageprocessing apparatus according to claim 1, wherein the incorrect-heightdetermining part inserts a flag as incorrect information into a portionwhere a difference not smaller than the predetermined value isdetermined to occur.
 9. The three-dimensional image processing apparatusaccording to claim 1, wherein the incorrect-height determining parttreats a portion where a difference not smaller than the predeterminedvalue is determined to occur, in a similar manner to a non-measurableplace.
 10. The three-dimensional image processing apparatus according toclaim 1, wherein the light projecting part projects structuredillumination for obtaining the distance image by use of at least a phaseshift method and a spatial coding method.
 11. A three-dimensional imageprocessing method for acquiring a distance image that includes heightinformation of an inspection target and performing image processingbased on the distance image, the method comprising the steps of:previously setting a first projection pattern and a second projectionpattern whose fringe direction is different from that of the firstprojection pattern as a plurality of projection patterns obtained bychanging a fringe direction of a projection pattern for structuredillumination, and projecting from a light projecting part incident lightas structured illumination of the first projection pattern from anoblique direction with respect to an optical axis of an image capturingpart; acquiring reflected light that is projected by the lightprojecting part and reflected on an inspection target, to capture afirst pattern projected image in the image capturing part; generating afirst distance image in a distance image generating part based on aplurality of images captured in the image capturing part; projectingfrom the light projecting part incident light as structured illuminationof the second projection pattern from the same direction as the firstprojection pattern with respect to the optical axis of the imagecapturing part; acquiring reflected light that is projected by the lightprojecting part and reflected on the inspection target, to capture asecond pattern projected image in the image capturing part; generating asecond distance image in the distance image generating part based on aplurality of images captured in the image capturing part; and makingcomparison in height information of a corresponding portion of theinspection target between the first distance image and the seconddistance image, to determine height information of a portion where adifference not smaller than a predetermined value occurs as incorrect ifthis exists.
 12. A three-dimensional image processing program which iscapable of acquiring a distance image that includes height informationof an inspection target and also performing image processing based onthe distance image, wherein the program allows a computer to realize: aprojection pattern setting function of previously setting a firstprojection pattern and a second projection pattern whose fringedirection is different from that of the first projection pattern as aplurality of projection patterns obtained by changing a fringe directionof a projection pattern for structured illumination; a distance imagegenerating function of generating a first distance image and a seconddistance image based on a first pattern projected image and a secondpattern projected image captured by projecting by a light projectingpart incident light as structured illumination of the first projectionpattern and the second projection pattern set by the projection patternsetting function, from an oblique direction with respect to an opticalaxis of an image capturing part and acquiring reflected light reflectedon an inspection target; and an incorrect-height determining function ofmaking comparison in height information of a corresponding portion ofthe inspection target between the first distance image and the seconddistance image, to determine height information of a portion where adifference not smaller than a predetermined value occurs as incorrect ifthis exists.
 13. A computer-readable recording medium or a recordingdevice in which the three-dimensional image processing program accordingto claim 12 is recorded.